Yes, this is completely possible and also reasonably common.
IP over IB (IPoIB) is well supported in several OS's, including Linux (Red Hat/CentOS/SL/Debian/etc), Windows, VMware ESX, and Solaris. Communication between OS's using IPoIB works very well, since (after the Infiniband drivers are installed) the OS's just see the traffic like normal IP.
(actually, that's a bit of a simplification, as you can go into the Infiniband settings on your OS's to tweak things for extra efficiency if you want, but the concept of "it's just like normal IP, but runs over a different physical layer" is correct)
There are also other Infiniband protocols (not just IPoIB) which take advantage of Infiniband's special properties more. Mostly to make use of RDMA. Things like SRP (a block storage protocol), iSER (iSCSI protocol enhanced for RDMA), and so on. These other protocols aren't as widely spread across all versions of all OS's as IPoIB, so some combinations work well, others not so much. But they're worth knowing about and looking into, depending on what you want to use Infiniband for.
Does that help?
(note - edited for typo fixes and some minor wording improvements)