I'd say that the crying factor would tell you, but at 18 mos. my nephew broke his fibula and didn't cry too much, he just insisted on sitting on the sofa and not moving (VERY unlike himself). My poor SIL didn't take him to the doctor for 3 days. She felt like a heel. But he barely cried.
When I broke both my collar bones at age 6 and when I broke all 3 bones in my left arm at age 12, I cried enough to make Sally Field envious.
Swollen and discolored meant a broken for me. But since there was nothing they could do except tape broken toes, it was a wasted trip to the doctor. I'm not sure what they'd do for a broken finger besides tape it, especially when it is such a tiny little 2 year-old pinky.
At the drugstore they have metal splints for fingers/toes with foam inside, and then you just tape it straight. It really hurts, but you don't have to be a rocket scientist to set a finger or toe. When I broke my toe the second time, I set it myself. Just make sure he gets some tylenol and TLC. He's so young it will heal in no time.