That is a fun question. Of course if you bring your baby naked in the cold weather, he will suffer from it, but that is the same thing for adults.
You only have to get proper winter gear, just as you would for yourself.
As a Canadian who is used to go outside at -10C to -20C every year and I experienced -40C. I would not be worried to bring a baby outside, but I would be more worried about getting a baby to sleep in a cold Japanese house. Were I live, there is law that require appartment ower who heat their building to maintain a minimum temperature of 19C.
Just properly dress your kid and it will be just fine.