Hello,
Did you install Sybase client components on the SQL Server machine? When running server-side data migration it will create a SQL Agent job that runs on the SQL Server machine and reads data directly from Sybase server, thus it needs to be able to connect to Sybase from that machine. Also make sure that client components you install there are of the same architecture as the server-side data migration engine you are using (by default it's 64-bit, but can be changed in project settings).
You may also be able to see more information in the log file, just make sure to switch all log levels to debug (Tools -> Global Settings -> Logging). Although if there is something happening on the server side, we may not log it in the client-side log.
Check Event Viewer on the SQL Server machine to see if there is anything being logged there.
Ultimately, if client-side migration works - I don't see any reason to invest in setting up server-side.
Regards,
Alex.