What's the difference between H1-B transfer and application for a company?

Techniquely, there is no such thing as H1-B. Every application is a new application. But if you are holding a H1-B with a company, you don't need a quote and can start working before new application get approved

If a company doesn't sponsor H1-B application, is it possible that it sponsors H1-B transfer?

I don't think so.

For transfer, do they need to pay for it?

somebody got to pay for it, either you or your company. The fee includes attorney fee and it was said there is new fee for INS.