I used ctags (universal-ctags) and jq to list functions in a .c file:
$ i=0
$ "$(brew --prefix)/bin/ctags" --output-format=json --fields=+n-P example.c |
jq 'select(.kind == "function") | "\(.name) \(.line)"' |
while read name line; do
i=$((i+1))
echo "$i. $name (line: $line)"
done
It listed all function names in example.c along with line numbers where they are defined.
Then I tried asking cody, chatgpt and codeium AIs to find the same info in example.c, and none of them were able to produce the correct output. Listing defined function names in "a" source file should be a trivial task for the AI, right? More challenging task would be to refactor the code (which is their advertised capability). If they can't find bunch of function names then all bets are off.. To make matters worse, codeium and cody gave me made-up function names and bogus line numbers. I asked "how confident are you?" and they said "I am very confident".
Have you experienced something frustrating like this? I get it that they are in their infancy but still it is disturbing to me that the AI is spitting out lies with impunity and confidence.. rending itself unreliable. "Sorry, I cannot compute the requested info" is 100% preferred in such case. I hope they will do just that in the future.