In a code language, TRY is written as RTY and GROUPS is written as GOPRSU. How will EXTEND be written in that language?

The logic here is sorting the letters of the word in alphabetical order.
TRY→Alphabetical order: R, T, Y→RTY

GROUPS→Alphabetical order: G, O, P, R, S, U→GOPRSU

Applying the same logic to EXTEND:

Letters are: E, X, T, E, N, D

Arranging alphabetically: D, E, E, N, T, X → DEENTX.