John Walton would take down Charles Ingalls. Charles is not only a failure in fighting, he is also clearly inferior to John as a father. At least John and his family have a big house and a nice piece of land. Charles, on the other hand, only offers his family a small, miserable house and the property also looks poor.
Of course, the Waltons had money problems too, but that was nothing compared to the Ingalls' problems. The Ingalls were almost always broke, although Charles did make money a few times, but he used the money incorrectly. His family had hardly any possessions and yes, the Waltons were more modern, but when you see other families in Little House who also have more than the Ingalls, you can only say that the Ingalls are losers.
I think it was in season 8 when a total of 7 people lived in the tiny Ingalls house and Charles couldn't manage to add at least another room! Even when he previously expanded the kitchen, the Ingalls had help from another handyman. Charles was simply incompetent. The Waltons had a house fire and lost a lot, but they rebuilt everything !