Where does “matter of fact” come from?

matter of fact (English): a fact; a more factual correction; something...