Terriers boss David Wagner was weighing up an £8million bid for the England Under-20 star who is keen to continue his development by playing regular football.
Southampton were also alerted to his availability, but Brown has decided to link up with Chris Hughton on the South Coast.
The Premier League champions are keen to hold on to Brown who has two years on his current contract and want to reward his progress with a new deal and pay rise.
Brown moved to Chelsea from West Brom aged 16 in 2013 and has had loan stints at Vitesse Arnhem, Rotherham United and Huddersfield Town.
The England Under-21 star enhanced his reputation scoring 26 goals last term for Bristol City in the Championship, and his parent club will be fascinated to see how he shapes up this season in the top flight with Swansea.
Meanwhile Kasey Palmer has agreed a new deal and has joined Huddersfield on loan, and England Under-20 star Jake Clarke Salter is expected to have a season on loan in the Championship.
The moment Huddersfield Town gain their first ever promotion to the Premier League
Leave a comment