I think I managed to find an answer that I like, I created the following class by inheriting from EqualityComparer:
Public Class ArrayComparer(Of T) : Inherits EqualityComparer(Of T())
Public Overrides Function Equals(x As T(), y As T()) As Boolean
Return x.SequenceEqual(y)
End Function
Public Overrides Function GetHashCode(obj As T()) As Integer
Return obj.Length.GetHashCode()
End Function
End Class
And then I just used .Distinct(New ArrayComparer(Of Integer)()) in my code. I knew there had to be a simple way to do it, I think this was just one of those first time things (and I have always been somewhat confused by the GetHashCode method). But this seems to work well enough (at least for my scenario).