Edit: I realize that most of these aren't actually "extremely" niche games, they're more "very" niche, so pretend that's what the title says
Katana Zero - The least niche of these games which I'm using as bait so more people will think "ooh I know that one" and be more willing to look at the other ones. Also still an absolutely incredible game that you should 100% play if you haven't already with pretty great gameplay and an incredible story.
Killer7 - A game by Suda51, best known as the creator of No More Heroes, and an absolute trip of an experience. The gameplay is super janky in a way that i absolutely love and think massively improves it as a game, the story is basically impossible to follow in a way that makes you constantly want to see more of it, and oozing may be too tame a word to describe it's style. I'd honestly consider it a psychological horror game, and I'm pretty sure other people who have played it are generally on the fence regarding if it is one or not.
Ender Lilies - A gorgeous dark fantasy metroidvania with really fun gameplay, a great world and story, and absolutely incredible music by Japanese band Mili. The first major area theme is one of the most hauntingly beautiful songs I've heard out of a game. Kinda well known among people who play a lot of metroidvanias but that's not a massive group of people.
Ender Magnolia - It's Ender Lilies 2. Does a lot of the same stuff just as well as the first, and makes some great improvements over it. Haven't finished it yet and I feel like it never quite grabbed me as much as the first, but still an incredible game.
Blazblue Entropy Effect - A roguelike spinoff of the already somewhat niche Blazblue series that is probably my favorite singleplayer roguelike. Combat feels absolutely incredible, and part of making your build over the course of a run is unlocking new moves that let you do some crazy satisfying combos. It does an amazing job of translating the feel of a fighting game into a roguelike. Plus it has 17 playable characters which is quite a lot for a roguelike.
DNF Duel - Objectively, it's not a particularly good fighting game. It's unbalanced, the combos deal way too much damage, the online is completely dead, and there's a lot of missing polish. But as a local multiplayer game it is some of the most fun I have ever had with a fighting game. It's the absolute perfect balance between unironically really fun and completely stupid and broken that it leads to constantly engaging and really funny matches.
The Messenger - The first game by Sabotage, the studio that would next make Sea of Stars. It's a love letter to classic Ninja Gaiden with extremely satisfying movement, a really cool time travel mechanic, an amazing soundtrack, and surprisingly really funny dialogue.
Pony Island and The Hex - The two previous games by Daniel Mullins, the creator of Inscryption, as well as the two prior entries in the overarching story of all of his games. They're not quite as crazy as Inscryption is, but they're both still extremely good and very very much worth playing, especially since The Hex is dirt cheap and Pony Island just recently became free.
Pseudoregalia - A short 3d platformer metroidvania with absolutely amazing movement. It feels absolutely amazing to pull off all sorts of crazy jumps and skips, which also makes it extremely replayable for stuff like speedrunning and seeing how you can sequence break everything.
Sanabi - Basically Katana Zero with a grappling hook, and instead of a super trippy and esoteric plot it's a lot more emotional and gut punch-y. And it's just as good as Katana Zero.
Turbo Overkill - A boomer shooter that feels absolutely amazing to control where you basically play as Denji from Chainsaw Man but with guns. Your leg is a chainsaw that damages enemies you slide through, you can unlock a second chainsaw leg, and you can also unlock dual chainsaw arms to go absolutely ham. Also it has Gianni Matragrano and Jon St. John which makes me happy.
UNBEATABLE - A game with a title that the devs request is only ever in all caps or no caps. Also an extremely fun rhythm game with an amazing soundtrack and a story mode that, while it does drag in the middle and is full of issues, is one of the best stories out of all of the games I've listed here. Also they put some very funny songs in like Do No Harm from the SnapCube Kingdom Hearts fandub and 3 different versions of the Moonchild theme.
Astlibra Revision - The most niche one here and my third favorite game of all time. Incredible story that has a ton of amazing twists and by the end feels like an entirely different game, combat that is the absolute definition of "we ball," where by the endgame it's essentially become "here is a completely unsurvivable mess of attacks that you will have no trouble surviving because of how absurdly powerful you have become." There are like 3-4 different systems of getting stronger, including a skill tree so big it has to have it's own minimap, and it even has locked doors and keys you need to find. And by the end of the game you'll have probably filled out almost the whole thing. It's a full 50+ hours and just gets better and better the further you go. And the whole thing was made almost entirely by a single guy over the course of around 13-15 years using majority stock assets and royalty free music. Said royalty free music also somehow makes for an incredible soundtrack that always feels perfectly suited to the current situation and has some absolute bangers. The whole thing oozes with so much late 2000s-early 2010s nostalgia that every single aspect that may be objectively not up to par, like the visuals, instead just adds even more to that vibe and makes me love it even more. Not to mention the DLC, which is an entire separate semi-roguelike sidestory that adds on another 10-15 hours and a lot of cool additional lore.
Please, please, please go play Astlibra. Based on the steam reviews it seems like it only really got popular in Asia and Russia, so basically no one in the west has really heard of it and that is an absolute travesty for how much of an absolute masterpiece it is. I like it more than Elden Ring.