Rep. Jesse Jackson Jr.’s extended leave of absence and eventual resignation from Congress may have tried his constituents’ patience, but now it may be affecting their pocketbooks as well.

According to a report from ABC News, the congressman’s departure and the special election to replace him may cost taxpayers roughly $5.1 million.

Illinois will have to hold two special elections to fill Jackson’s seat, first a primary and then the general. Jackson Jr.’s district is so heavily Democratic that the winner of that party’s primary would be heavily favored in the general.

This November, Jackson Jr. handily won re-election with 63 percent of the vote despite being deeply embroiled in an ethics controversy.

Jackson Jr. has been accused of indirectly offering money to buy President Obama‘s former Senate seat and using campaign funds to finance improvements on his home.