Hi, what values for a and b should I use? My program is working with the test vector, but using a=2 and b=3 for the given ciphertext is not working, Thanks!

PS: I was thinking a bit and should I "brute-force" it? But how will I know what the right plain text is?

a and b have different values than the test vector in the actual problem. In order to determine what the right plaintext is you'll have to figure out a way to programmatically identify english / valid plaintext.

@aaossa, you're on the right track. You'll need to find a way to determine the values of a and b programmatically. Since the block size is small and the size of the alphabet is small (29 characters), a and b are indeed brute-forcable. You'll need some way to determine if text is English or not. Try reading this article about frequency analysis and this article about using chi squared analysis to get started.